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Abstract 

To address the paucity of data on the use of MOOCs in “traditional” postsecondary institutions, Ithaka 
S+R and the University System of Maryland studied the feasibility of repurposing MOOCs for use in 
hybrid, credit-bearing courses. In this paper we will describe the design of a large-scale study undertaken 
to examine the use of MOOCs in fourteen campus-based courses, followed by two types of findings: 
First, we will share quantitative outcomes from students in hybrid sections, comparing students who took 
MOOCs with those who were taught in a traditional face-to-face manner; second, we will share 
qualitative findings on the opportunities and challenges presented by the use of MOOCs on campus. 
Finally, we will reflect on what would need to occur in order for these models to see widespread adoption 
in the future. 


Introduction 

When Massive Open Online Courses (MOOCs) first attracted widespread public attention in 
2011, some predicted that they would disrupt higher education by replacing traditional classroom lectures 
or even entire institutions. However, as it became evident that these extreme proclamations were not 
going to become reality, some investigators turned to more subtle questions of how MOOC content might 
be incorporated into traditional degree programs and environments that serve mainstream American 
college students in order to improve student outcomes and/or reduce costs. 



One of the key selling points of MOOCs is the large volumes of data they generate about learners, 
providing instructors with a better understanding of learner behavior and informing course design. 
However, these data only tell us about the people who take MOOCs, and evidence reported by edX 
indicates that most people who participate in MOOCs are not the same as those pursuing degrees in 
American colleges and universities (Ho et al., 2014,). Thus, the learner data from MOOCs tells us little 
about how typical postsecondary students would fare in these courses. 

Some MOOC creators and institutions have experimented with using MOOCs in their existing 
courses to replace or enhance the lecture material. MOOCs offer high-quality videos from some of the top 
professors and institutions worldwide, currently at no cost. Therefore, they could be seen as an alternative 
to creating one’s own lecture videos or as a way to convey knowledge on a subject beyond the instructor’s 
own expertise. That said, little is known about how instructors can actually incoiporate these materials in 
their classes, the potential for them to improve the classes, the challenges they bring, and how students 
respond to them. 

Literature Review 

A case study from Vanderbilt University found a fairly positive response from students when a 
computer science MOOC from Stanford University was embedded in a graduate seminar (Bruff, Fisher, 
McEwen & Smith 2013). An experiment at San Jose State University found substantial gains in pass rates 
for students in sections using an engineering course from MITx compared to the traditional sections of the 
course (Ghadiri, 2014). However, some differences in the sections and limits on student background data 
lessen the weight of these findings. Many components of the course were modified in the sections using 
the MOOC, and students had the same amount of class time. Moreover, detailed analysis of the 
background characteristics is not available, so we cannot be sure that there were not significant 
differences in the baseline student populations. Another experiment at San Jose State University (SJSU) 
with Udacity and introductory math courses found mixed results. Pass rates in online sections were very 
low, but researchers attributed this at least in part to the targeted student population (Firmin, Schiorring, 
Whitmer, Willett & Sujitparapitaya, 2013). They found that students who were matriculated at SJSU were 
more likely to earn passing grades than nonmatriculated students. Overall, they concluded that the 
academically at-risk student population involved in this study fared better in face-to-face courses, but still 
found room for optimism about the potential of the online resources. While these studies provide insight 
into how MOOCs can be used and some of their impacts on students, each is about a very specific case 
with a set of qualifications that make it hard to generalize the results to other instructors and institutions 
that may be able to take advantage of MOOCs in the classroom. 

There is some robust research on the use of other online learning materials for undergraduate 
students that suggests that these types of materials do not appear to harm student outcomes when used in 
hybrid formats, though certain subgroups of students may perform worse in fully online courses. Bowen, 
Chingos, Lack, and Nygren (2014) studied the use of Carnegie Mellon’s Open Learning Initiative 
Statistics course in hybrid undergraduate classes and found no significant difference between the hybrid 
and traditional sections in student pass rates and assessment scores, including for underrepresented 
minorities, students from low income families, and first-generation college students. Figlio, Rush and Yin 
(2013) found that student learning outcomes in a traditional lecture-style course were not significantly 
different than those for students who had access only to the online videos of the same lectures, but they 
did find that the online-only treatment had negative impacts for some subgroups of students (Hispanics, 
males, and lower achieving students). 

Research focused on community college students and entirely online courses finds that students 
perform worse in online courses than face-to-face courses, and that the gap in performance between these 
two modes is larger for some underrepresented minorities, males, and students with lower levels of prior 
academic achievement (Jaggars, 2012; Johnson & Mejilla, 2014; Xu & Jaggars, 2013). It also finds that 
the performance gap between students in online and face-to-face courses varies by subject, with online 
courses having the most negative results in social science and professional studies courses (Xu & Jaggars, 



2013). Findings about the impact of online courses on progression toward an associate degree or 
transferring to a four-year institution are mixed. A review of the literature by Jaggars in 2012, as well as a 
report by Jaggars, Edgecombe, and Stacey (2013), suggests that online courses may hurt progression 
toward a degree; however, more recent work finds that taking online courses improves a student’s 
likelihood of earning an associate degree or transferring to a four-year institution (Johnson & Mejilla, 
2014; Shea & Bidjerano, 2014). While these studies focus on different types of courses (online instead of 
hybrid) and different populations of students than our study, they point out important findings for 
disadvantaged students that deserve further exploration. Unfortunately, much of the online-learning 
research lacks careful comparisons of students in different course formats in order to determine the true 
effect of these materials (Means, Toyama, Murphy, Bakia & Jones, 2010). 

Another important issue is what effect course design and implementation have on student 
outcomes. Jaggars and Xu (2013) developed an online course-quality rubric and explored the relationship 
between each quality area and student end-of-semester performance in 23 online courses at two 
community colleges. The results indicate that the quality of interpersonal interaction within a course 
relates positively and significantly to student grades. Additional analyses based on course observation and 
interview data suggest that frequent and effective student-instructor interaction creates an online 
environment that encourages students to commit themselves to the course and perform stronger 
academically. Again, this study pertains only to online courses; not hybrid formats. 

In addition to understanding the impact on student outcomes it is also important to consider 
whether and how faculty members actually make use of online course materials. A study by Bacow, 
Bowen, Guthrie, Lack and Long (2012) provided insight into the banders to adoption of online learning, 
benefits universities anticipate from these technologies and how they can be molded to fit institutions’ 
needs. The main findings were the need for open and shared data on student learning that can be 
generated from online learning technologies and the need for sustainable and customizable platforms for 
delivering online learning instruction. This study was conducted before the creation of MOOCs, but we 
find many of their findings directly relevant to our experiences with institutions and faculty. 

Since millions of dollars continue to be invested in the development of MOOCs, producing 
hundreds if not thousands of high-quality online courses, we think it is important to understand how these 
courses might be able to benefit mainstream undergraduate students. In partnership with the University 
System of Maryland (USM), and with extensive cooperation from Coursera, a team of researchers at 
Ithaka S+R, a nonprofit research and advisory service focused on higher education, set out to learn more 
about potential uses of MOOCs in campus-based settings. The questions we wished to address included 
the following: How does the use of MOOCs in hybrid courses impact student outcomes? Can these 
materials be used by faculty members who did not create them? What kinds of implementation challenges 
arise, and how can they be overcome? How does use of these tools impact time to plan and teach a 
course? 

This paper will describe the study design and data collection methods and then will move to 
findings in terms of student outcomes and student reactions. Finally, we will briefly discuss the 
implications of these findings and topics for further research. 

Method 

There were two main components to this mixed-methods study: a set of four side-by-side tests 
using MOOCs in introductory hybrid courses, and 10 case studies. (Three additional side-by-side tests 
using the Introductory Biology course from Carnegie Mellon’s Online Learning Initiative were also a part 
of the larger study, but we do not include them in these results so that we can focus our analysis on 
MOOCs.) For the side-by-side tests, we compared outcomes of students in hybrid sections using MOOCs 
with outcomes of students in traditionally taught sections of the same course. Students were not randomly 
assigned to different sections: we controlled for student background characteristics, including age, gender, 
race/ethnicity, year of study, SAT/ACT scores, parental education and family income when analyzing the 
outcomes (pass rates and common assessments). The case studies explored the use of MOOCs in a variety 



of course formats, disciplines and divisions. For these we did not attempt to measure outcomes, as there 
was no baseline cohort for comparison. 

Procedures 

In order to recruit test cases, Ithaka S+R and USM staff presented the project at each campus to 
groups of faculty and administrators. We solicited “statements of interest” from faculty members, asking 
them to declare their interest in participating, identify a course on campus and a MOOC that might work 
in that context, and describe a goal or problem that might be addressed through use of the online content. 
These statements were vetted based on several criteria: whether the proposed use of the MOOC was 
substantial enough to illuminate the research questions, whether there was a compelling reason for the use 
of the MOOC, and whether we could obtain permission to use the desired MOOC. This required some 
back-and-forth with Coursera and its partners. Sixteen courses were selected to be part of the study, from 
an initial response of 30 statements of interest. Two of the courses were later eliminated from the study 
because we were not able to use the MOOCs upon which they relied. 

It is important to note that our group of instructors was self-selected and probably more motivated 
to teach with online technologies than the average instructor. This selection of instructors may have led to 
more positive reviews of the experience as well as better outcomes than if the instructors (and courses) 
had been randomly chosen. 

Participants 

Fourteen faculty members who were interested in teaching with MOOCs were recruited from 
seven institutions across the USM, which included research universities, regional comprehensives, a 
metropolitan institution, and historically black universities. In all but one instance, Coursera set up local 
versions of MOOCs, which the USM faculty could customize to meet the needs of their courses and 
students. In four instances we set up side-by-side tests comparing outcomes of students in traditionally 
taught sections with those of students in hybrid sections using MOOCs. Three of these had reduced class 
time. 

In addition, we conducted 10 case studies, in which faculty incorporated MOOCs into a variety of 
small, single-section courses. The tests covered a diverse range of disciplines, including math, statistics, 
computer science, communications, studio art, poetry, philosophy, nutrition, ancient Greek and Roman 
mythology, and introductory literature. Sizes ranged from six students up to 60. Instructors used MOOCs 
to replace some of the course content or enhance the existing course materials. One of these courses was 
entirely online, and in another students enrolled in the public offering of the MOOC and engaged in 
parallel seminar discussions and assessments at their own institutions. Several case studies involved first- 
year learning communities (linked courses that enroll the same cohorts of students and are intended to 
reinforce each other). 

In total, there were 855 students involved in these tests, including 664 in side-by-side tests and 
193 in case studies. We obtained background data for 836 students, and 562 completed final surveys 
(including those in control sections). We conducted interviews with 19 instructors, some of whom taught 
different sections of the same course. Tables 1 and 2 provide more details on the courses, students, 
formats, and outcome measures involved in both the side-by-side comparisons and the case studies. 

Data Collection 

For all tests we collected data on student backgrounds, enabling us to analyze survey responses 
for subgroups of students (such as first-year students). We also conducted student surveys in all tests as 
well as interviews with all instructors before the beginning of the term and after the conclusion in order to 
understand how they used the MOOCs in their courses, as well as their experiences and perspectives on 
what worked and did not work. Most of these data relate to the implementation challenges that arise in 
embedding MOOCs in campus-based courses. The interviews followed a standard set of questions, and 
we coded and tabulated categories of comments. 



Table 1 Courses and Students Involved in Side-by-Side Comparisons 

Number Minutes Minutes 

of Students in Students in per week per week 

treatmen treatment control treatment control Outcome 


Course t sections sections sections section met section met measure 


Statistics 2 

64 64 

150 

150 

Posttest; 

common 

final 

Pre-calculus 2 

55 67 

120 

220 

Posttest; 

common 

final 

Computer 4 

science 

92 84 

75-100 

150 

Common 

midterm 

Communication 4 

104 103 

80 

160 

Posttest; 

s 

Table 2 Courses and Students Involved in Case Studies 

Course Format notes 

Number 

of 

sections 

Number 

of 

students 

common 

final 

Minutes per 

week they met 

Art 

MOOC used for content portion 

1 

9 

160 

Poetry 

Live public offering of the MOOC 

1 

11 

150 

Genetics 

MOOC used to replace some 
lectures 

1 

60 

75-150 

Psychology 
learning community 

MOOC used to replace some 
content in two courses 

2 courses 

24 

250 

Information 

technology 

MOOC replaced middle third of the 
course 

1 

19 

0 or 150 
depending on 
week 

Political science 

Supplemental 

1 

11 

150 

English Learning 
Community A 

MOOC used as a common 
experience across three courses 

3 courses 

17 

210 

Philosophy 

Summer pilot, entirely online 

1 

6 

0 

Literature 

Two MOOCs used to provide 
greater insight into ancient 
mythology 

1 

17 

150 

English Learning 

MOOC used to provide context to 

1 

19 

150 


Community B the literature covered in class 


Side-by-side Comparisons 

Participating institutions provided administrative data related to academic preparation and demographic 
characteristics for all students in treatment and control groups. Students who consented to participate in 
the study were asked to complete surveys designed to solicit additional information about their back¬ 
grounds (e.g., parental education and family income), as well as interest in the course subject and their 
experiences in the course. Surveys were conducted in control sections as well as treatment sections. We 
allowed the institutions discretion over whether to offer small incentives to students for participation, and 
three tests chose to do so. 

Due to the range of courses involved, we could not implement a uniform posttest. Instructors for 
each course determined what their students should learn and designed assessments according to those 
standards. We asked treatment and control group instructors to agree on common exams or at least a 



significant subset of common questions on exams, and these are the primary outcome measures. One 
limitation is that these scores cannot be compared to a benchmark outside of the given university (or 
course) and are subject to the instructors’ discretion of what students should leam. A few instructors also 
mentioned that slight differences between content in the treatment and control groups may have 
influenced students’ scores (if some content on the assessments was covered more in one section than 
another) independent of how well students learned the content taught in their sections. We also obtained 
pass rates and grades, which can be subject to variations in grading standards among instructors. 
Although no single measure provides a perfect assessment of student performance, we believe that taken 
together these three measures provide a reasonable assessment of student performance. 

Case Studies 

We conducted 10 case studies using MOOCs in courses that did not allow for rigorous evaluation 
of learning outcomes due to smaller course enrollments. The purpose of the case studies was instead to 
deepen our understanding of the implementation process, to explore ways that MOOCs could be used to 
enhance teaching and learning in different types of courses, and to gain insight into student and instructor 
experiences in a larger set of instances. While instructors have views as to whether student learning 
bene fitted from the use of MOOCs, we cannot compare them to a reference cohort; thus, we do not report 
passing rates or grades for these courses. 

Findings 

This section reports quantitative and qualitative findings from both the side-by-side comparisons 
and the case studies in order to present a more holistic picture. More detail on the findings and data 
analyses for each test type can be found in the complete report (Griffiths, Chingos, Mulhem, & Spies, 
2014). 

Student Learning Outcomes and Survey Responses 

After controlling for SAT scores, cumulative GPA at the beginning of the semester, 
race/ethnicity, gender, parental income, and student age, differences between students in hybrid sections 
and those in traditionally taught sections were not statistically significant at the 10% level when averaged 
across the four side-by-side comparison courses (see Figure 1). In other words, the hybrid effect on 
student outcomes was not statistically different from zero. These outcomes are noteworthy in 
consideration of the fact that faculty members were teaching redesigned hybrid courses for the first time 
using new technology, and the fact that hybrid sections had an average of 103 minutes of face-to-face 
class time per week, compared to 166 minutes for traditionally taught sections. Moreover, we did not see 
evidence that academically at-risk subgroups of students performed consistently better or worse in hybrid 
sections, but rather a pattern of small positive and negative effects. 



Figure 1. Average posttest/exam scores and pass rates for hybrid versus traditional sections. 



On the other hand. Figure 2 shows that student ratings of the hybrid sections were significantly 
lower {p < .05) in terms of overall satisfaction and how much they felt they had learned. The impact on 
their interest in the subject and how difficult they found the course was not statistically significant at the 
10% level. (These ratings are on a five-point scale.) Moreover, students agreed more with the statement “1 
learned what 1 needed to in the course primarily from my instructor” than with the statement “I learned 
what 1 needed to in the course primarily from Coursera,” with respective ratings of 3.7 versus 2.9 (5 
means strongly agree while 1 means strongly disagree). 



Overall Rating Raised Interest Amount Learned Difficulty 


Figure 2. Student survey responses for hybrid versus traditionally taught sections. The differences for Overall 
Rating and Amount Learned are significant at the p <.05 level. 

Student ratings of case study courses were somewhat higher, with average overall ratings of M = 
3.5 and M = 3.4 for how much students felt they learned. We have no way to know whether these higher 
ratings reflect the success of these courses or the characteristics of courses involved in the different types 
of tests. On average, students gave lecture videos a score of M = 3.3 for educational value, compared to M 
= 3.5 for in-class activities. In side-by-side comparisons, these scores were M = 3.2 versus M = 3.8. In 
half the case studies, students gave higher ratings to the educational value of lecture videos relative to in- 
class activities, while in the other half we found the opposite responses. 

Costs 

One of the most important unanswered questions is whether and how online learning technologies 
can be used to reduce costs for institutions on a large scale. In this project we could not come close to 
addressing all the questions we would have liked to answer, but we were able to compile a baseline set of 
data for the time required to redesign and deliver a hybrid course using readily available online course 
materials. Faculty time spent on planning and teaching is one of the largest costs associated with a course, 
and we focused primarily on understanding how this time was spent and the impact of using online- 
learning platforms on amount of time spent. 

Faculty members reported investing considerable time redesigning their courses with MOOCs. 
The time savings that some instructors found during the semester were offset by high up-front time 
requirements. For the nine instructors who submitted records of the time they spent working on the 
course, reports ranged from 34 to 248 hours spent on course planning, with a median of 58 and mean of 
100 hours. This included time spent watching all the MOOC videos and reviewing assessments, figuring 
out how to put the pieces together, learning to use the platform, and actually setting up the online 
components of their courses. Since we did not have the opportunity to repeat the entire set of tests, we do 
not know how much time would be needed to plan and deliver these courses a second time. About a third 
of the participating faculty members thought they could save time in the future (see Figure 3). 
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Figure 3. Instructor responses in final interviews. 

A key unknown in future cost calculations is the license fee for MOOCs on Coursera. For the 
putpose of this study, Coursera’s partners gave permission to use their courses for free, and Coursera did 
not charge for its services in setting up and supporting local instances of the MOOCs. It is unlikely that 
these materials would continue to be available under these conditions. 

Faculty Teaching Approaches and Interview Responses 

Participating faculty members identified a number of opportunities for using the MOOC materials 
to enhance their courses. In all but one case, Coursera set up local instances of MOOCs, enabling 
instructors to tailor the selection and sequencing of course content, timing, and other elements to fit the 
needs of their courses. When asked what components of MOOCs they used in their courses, 13 faculty 
reported that they used multiple components, three faculty used the entire MOOC, and two used videos 
only. 

None of the instructors relied solely upon assessments from the MOOCs, suggesting that these 
were the element of the MOOCs least well calibrated to the specific needs of each course. All instructors 
used videos, and several used the embedded quizzes within the videos to track whether students were 
engaging with the online content. A couple of the instructors required students to submit screenshots of 
their interactions with the Coursera website to show that they had completed assignments. 

Opportunities for enhancement 

Faculty partners identified six types of benefits for students in their hybrid courses using 
MOOCs. These included the following: 

1. Replacing lectures. A pre-calculus instructor chose to use a MOOC in her class because she 
observed that teaching math at this level through lectures was not effective and she wanted to see whether 
assigning students to learn new concepts independently and using class time for active problem solving 
and group work would improve outcomes. In a similar vein, another instructor saw an opportunity to 
enhance an online introductory philosophy course with online lectures, as she did not have the time or 
resources to create these materials herself and felt that they would be highly beneficial for students. In 
both cases, MOOC videos delivered information in a manner similar to that of lectures but did not need to 
be created from scratch (though, as we saw, redesigning a course using a MOOC still took a lot of time). 



2. Augmenting or replacing secondary materials. One professor used two MOOCs on ancient 
Greek and Roman mythology in place of textbooks and other secondary sources. Students in the study 
saved money because they did not need to purchase a textbook (though, as noted above, we do not know 
how expensive license fees might eventually be for MOOC content or how they would compare to 
textbook prices). In a different course, an instructor mentioned that students were more likely to watch the 
videos than read the textbook because a number of students were unwilling to pay for a textbook. Another 
course used a MOOC to add macroeconomic context to an introductory course in comparative politics. A 
statistics course used a MOOC to supplement (but not replace) classroom instruction. 

3. Filling gaps in expertise. Two instructors were attracted by the possibility of using MOOCs 
to add expert voices that complemented their own areas of expertise, such as in interdisciplinary programs 
or survey courses. For example, one professor with a background in information technologies used a 
MOOC for a module of a human-computer interaction course on product design. 

4. Exposing students to other styles of teaching and class discussion. Several faculty members 
believed that MOOCs could benefit their students by exposing them to different styles of teaching and 
class discussion. For example, one taught a course on modem poetry and had her students participate in 
the live offering of a corresponding MOOC, in addition to holding weekly face-to-face class meetings. In 
the past she had struggled to engage students in critical analysis of texts rather than statements of personal 
likes and dislikes. The seminar style discussions in the MOOC videos provided a model for her desired 
style of intellectual discourse. Another team of professors embedded a MOOC on critical reasoning and 
argumentation into a first-year oral communications course in order to expose students to different 
teaching styles and personas. 

5. Reinforcing key skills, such as critical thinking. A studio arts instructor augmented a course 
on public art installations with a MOOC on critical reasoning and global challenges. This addition served 
the dual purpose of teaching students to approach art more analytically and providing a background on 
issues such as water management and obesity that could be subjects for their art projects. A second 
instructor integrated a MOOC on nutrition into a program seeking to strengthen basic life skills, such as 
sleep and eating habits, which have been shown to correlate with student success. 

6. Teaching students how to learn online. A professor incorporated a critical reasoning MOOC 
into a first-year learning community, using it both for its instructional content and as a vehicle for 
teaching students to become more effective learners. Students viewed MOOC videos together with an 
undergraduate mentor and discussed strategies for identifying core concepts and effective note-taking. 

Faculty partners also identified several ways in which they themselves benefitted from use of 
MOOCs. One benefit was professional development, as viewing MOOC content afforded an opportunity 
for instructors to see how other faculty members in their fields teach similar material and, in some cases, 
to learn new concepts. Another was relief from time pressure during the semester, as most hybrid courses 
had reduced class time. For example, the professor who used a MOOC to deliver a module of her course 
on design appreciated the time savings this approach provided during the semester, as she was teaching 
five different upper level courses at the time. (She found it helpful that she saw many of the same students 
in other classes during this period, enabling her to check in with them from time to time.) 

Finally, three faculty members liked the flexibility to use class time in different ways without the 
pressure to cover content. For example, a sociology professor, who by his own admission had been slow 
to adopt instructional technology, reported feeling “excited” and “exhilarated” by the possibility that 
using MOOCs could enable him to reorient the course from a focus on teaching content (which students 
could learn online independently) toward “developing a sociological imagination” through project work. 
Fie described the technology tools, such as peer review, as “incredibly empowering” for students and felt 
energized by the opportunity to overhaul the way his course is taught. (This test is not formally part of the 
MOOCs study because it ended up using Pearson’s MySocLabs in the hybrid sections, but the MOOC 
that was initially under consideration helped to spark many ideas about how the course could be 
redesigned.) 



Implementation Challenges 

The participants in this study worked through many implementation challenges in order to deliver 
their courses. Most of these stemmed from the fact that MOOCs on Coursera—as well as the platform 
itself—were not originally designed to be repurposed for use in other professors’ classes, so many 
modifications were necessary to fit what sometimes felt like “square pegs in round holes.” The primary 
issues encountered were content fit, intellectual property, and technology integration. 

1. Content fit. Finding and adapting online content for a hybrid course posed the greatest 
challenge for faculty partners. MOOCs reflect the priorities of their creators, and these are not necessarily 
the same priorities that other faculty have for their own students. Moreover, academic departments 
develop degree program curricula as a whole, making deliberate decisions about when and where certain 
content should be taught and which competencies should be assessed within specific courses. To integrate 
a MOOC into an existing class is not necessarily a simple case of choosing what pieces to include or 
exclude. Even with online course materials that are a fairly close fit with the pedagogical approach of the 
instructor and needs of the students, the local instructor may need to reconceptualize or restructure his or 
her existing course to fit with the online content. 

About a third of the faculty partners said that the MOOC assumed too high a level of prior 
knowledge or quantitative skills, while four thought that the assessments did not demand the level of rigor 
instructors would expect for courses in their institutions. Some faculty members reported that the MOOCs 
they were trying to use taught concepts in a different way or emphasized different topics than they did in 
their own approach to teaching a subject. In one course, for example, the USM instructor found that some 
MOOC units covered topics that would be taught in upper division or graduate courses at his institution. 
Another found that the lecture videos presented topics in a different sequence than his syllabus, and 
moreover, that they taught concepts in a cumulative fashion with frequent references to previous units, 
making it difficult for him to use them out of order. He ended up creating most of the online videos for 
his hybrid sections and only used around 10% of the MOOC videos. (This was in a pilot conducted during 
the spring of 2012 and is not otherwise reported here.) 

Content fit was particularly important in introductory courses, which are integral to the 
curriculum and need to be well matched to the abilities of the student population and requirements of 
subsequent courses in those programs. Finding a MOOC that was a good fit with an existing course was 
fairly hit or miss given the small number of options available at the time, and even MOOCs that were 
reasonably well suited required some customization or “bridging.”_As a result of these challenges, seven 
faculty members felt after the study that they could create more appropriate online course materials for 
their students. All of the instructors participating in this study created their own assessments to go with 
the MOOCs. In general, we learned that the first generation of MOOCs were structured as coherent 
narratives, while professors using these materials preferred more flexible modules or “chapters.” 

2. Intellectual property. During the course planning and delivery period for this study, there was a 
lot of uncertainty around ownership of MOOCs. In particular, there were questions about whether the IP 
should belong to the faculty member who created it or to the home institution. Additional questions arose 
for which there were no clear answers or precedents, such as the following: When an instructor adapts a 
MOOC for use in a hybrid course, who owns and/or has rights to use the adapted version? What kinds of 
adaptations should be permitted? Does the instructor who adapted the MOOC have any assurance that 
these materials will be available for future use? If a new version of a particular MOOC is released, will 
the instructor be required to use the most recent version, even if that means making adaptations to his or 
her course all over again? These are critical questions in view of the extensive efforts required to redesign 
courses using the online materials. For this study we devised ad hoc solutions—and were fortunate to 
have partners who were willing to participate despite considerable uncertainties as to how these issues 
would be resolved. It is clear, though, that more standardized policies are necessary to enable large-scale 
use of these materials in hybrid formats. 


A critical underlying issue is how to balance the instructors’ freedom to teach their courses as 
they wish with the desires of some MOOC creators to monitor or restrict the way their online content is 
used. The unique value of MOOC materials derives at least in part from their authorship by exceptional 
professors, and we detected understandable anxiety from several of those approached for this study about 
losing control of their intellectual creations. Two of these professors expressed concerns about the idea 
that their courses might be “canned” or repurposed by unknown faculty in other institutions. These 
worries were by no means universal, however; the majority of Coursera’s partners whose permission we 
sought to include their courses in our study consented readily. 

While MOOCs are “open” in the sense of being free to students who enroll in public offerings, 
they have fairly restrictive terms for other kinds of use. One may not take these courses as part of any 
kind of tuition-based or credit-bearing course without specific permission, nor is one allowed to modify or 
repurpose the content in any way. There is currently no standardized way to obtain these permissions. 

3. Technology integration. Integrating online learning platforms with campus technology 
infrastructure presented another set of challenges. The MOOCs used for this study did not “plug” easily 
into local learning management systems, and thus extensive efforts were required by Coursera and USM 
staff to support faculty and to enable students to access their local versions of the online course content. 
Instructors had to work through questions such as the following: Should they embed links to individual 
videos and online assessments in the campus LMS course page? Should they simply post one link to 
Coursera and have students navigate their courses within that platform? Or should they download and 
embed all desired content from Coursera directly into the campus LMS so that students could find all 
course materials in one place? 

Even after these decisions were made, technology integrations did not always go as planned. 
Although mechanisms were in place in all but one campus to enable “single sign-on” to the online 
platforms, many students encountered problems accessing the appropriate versions of the MOOCs. One 
team of instructors that had planned to use an integrated authentication approach abandoned this solution 
once the course got underway. In survey responses, 21% of free-text comments about what could be 
improved in the course concerned technology. 

The Coursera platform was not intended to be used by instructors on campus, and thus the 
features did not meet several of their needs. For example, it was cumbersome to transfer grade 
information to the local LMS systems, and the peer review feature did not work well for small groups of 
students. One instructor observed that the Coursera dashboard did not work well for a campus-based 
course, as she could not monitor the progress of individual students. 

Finally, these technology integration issues resulted in incomplete usage data. Many students 
accessed Coursera through multiple accounts or in other ways that prevented us from keeping close track 
of individual usage. While data from some courses may be complete, uncertainty about how students 
were accessing the system mean our data are generally unreliable. 

4. Faculty experience. Overall, the faculty members who participated in the study gave 
overwhelmingly positive reviews of their experiences. Of 19 instructors who participated in final 
interviews, 15 said they would like to teach again using MOOCs, and an additional three said they might 
under certain circumstances. Roughly the same numbers said they would recommend this approach to 
teaching courses to their colleagues. These responses are not entirely surprising given that most of them 
had volunteered to participate in the study. Still, it may be indicative of some of the benefits instructors 
found in using MOOCs that were discussed above. 

Overall Assessment 

About three quarters of the instructors involved reported that they felt they achieved what they 
had hoped in their courses using MOOCs. Those who wished to expose students to different styles of 
class discussion were particularly positive about what they observed in their classes. For example, the 
instructor of the modem poetry course thought her students pushed themselves to make more analytical 
comments than in past offerings of the course. Tier favorite part of the course was a series of sessions in 



which she videotaped class discussions with students acting as facilitators, thereby producing her own set 
of MOOC-like videos. Another instructor reported that her students demonstrated a stronger 
understanding of how to build an argument than she had seen in the past. Instructors who taught an oral 
communications course reported that students took time to get used to the styles of professors teaching a 
MOOC, but they saw this as a valuable experience for students who were accustomed to a homogenous 
learning environment. 

A professor who used two MOOCs to replace secondary source materials was extremely pleased 
with the results. He thought the MOOC videos provided a more accessible and engaging way to deliver 
technical information than textbooks. His perception was corroborated by surveys, as his students rated 
the lecture videos as the most valuable component of the course. 

Experiments were least successful when the topic of the MOOC was not closely linked to the 
campus-based course. One instructor had hoped to broaden the scope of his course by incoiporating a 
MOOC that was not tightly coupled and students appear not to have engaged with either the in-class or 
online components of the course. In another case study, one student wrote in a tree-text response saying 
she did not understand how the MOOC related to other parts of the course. 

We also did not find any benefits to students in a statistics course that supplemented traditional 
instruction with a MOOC without reducing class time. The instructors observed that students did not 
expect material covered only in the MOOC to be included on tests, and as a result they skipped over it. 
One instructor speculated that students approached course content from a utilitarian perspective: If 
watching all the MOOC videos counted for a small number of points relative to less time-consuming 
activities, they focused their efforts elsewhere. 

Limitations 

Our methodology carried some important limitations. Perhaps the most important is that we were 
not able to repeat the side-by-side tests a second time to see whether instructors might obtain better results 
with more experience. Similarly, these tests were conducted with MOOCs available in summer 2013. It is 
reasonable to expect that the technical and pedagogical quality of these materials has improved with time 
and experience. In the future, we recommend conducting tests such as these over multiple iterations to 
provide formative as well as summative findings on the use of these technologies and to assess benefits to 
students and faculty over time. 

In addition, we did not attempt to randomly assign either students or instructors to treatment and 
control sections. Our detailed analysis of background characteristics (available in the full report) indicates 
that the two cohorts of students were extremely similar and gives us some confidence that selection bias 
was not an important factor driving our results. While we attempted to control for some instructor 
background characteristics, such as academic rank, it is possible that treatment group instructors differed 
in unobservable characteristics, such as motivation. As noted earlier, the posttests were created by local 
faculty in the Maryland system in accordance with the demands of their programs. We were not able to 
identify suitable nationally recognized instruments with which to validate their assessments of learning 
outcomes. 

Finally, it would be very difficult for others to replicate the implementations used in this study. 
Instructors were given considerable latitude to design their courses using MOOCs, and the case studies in 
particular involved an idiosyncratic mix of course topics and contexts. In future work it would be 
beneficial to narrow the range of variables in order to provide more conclusive evidence about the impact 
of key design decisions. 

Conclusion 

This study finds that MOOCs can be used to replace some amount of class time without harming 
students in terms of test score and pass rates. We found that students in the hybrid sections relied heavily 
on the time they did have with professors, and that those in side-by-side tests reported lower levels of 



satisfaction and felt that they learned less. This finding is consistent with other large-scale studies (Bowen 
et al., 2014; Figlio et al., 2013; Joyce, Crockett, Jaeger, Altindag & O’Connell, 2014) and may suggest a 
higher perceived value of classroom time. The variation of student responses across our set of test cases 
suggests that student reactions to hybrid formats may depend on the nature of the course, the quality of 
the online materials, and how effectively they are integrated into the course. 

Our qualitative analysis indicates that there are a number of potential benefits for both students 
and faculty members associated with use of MOOCs, including flexibility in the use of class time, 
enhanced critical analysis skills, and exposure to different types of teaching styles and class discussions. 
Surveys indicate that students found value in the online resources, though slightly less than in the in-class 
activities. One instructor argued strongly that faculty members should take advantage of all tools and 
resources available to them to improve education given all the challenges they face with students. Most 
participants said they would like to teach again with MOOCs and would recommend this approach to 
their colleagues. 

At the same time, the conditions are not yet in place to enable adoption of these models on any 
significant scale. We do not see signs that MOOC providers (either platforms or content creators) are 
developing business models to support institutional licensing, and we observe that intellectual property 
provisions still prohibit use of MOOCs in credit-bearing courses without special permission. Technology 
integration is still a largely manual process. Any faculty member who did have an interest in using these 
materials in his or her course would be hard pressed to figure out how to do so. Moreover, few 
universities provide incentives or support for faculty members to use MOOCs—or other types of existing 
online courseware—in any systematic way. 

The path of least resistance seems to be for MOOC providers to serve unaffiliated learners and for 
faculty members teaching blended courses to rely upon other types of online resources or to create online 
materials themselves. From a system-wide perspective, this outcome would be unfortunate given the 
resources being invested in MOOCs and our national imperative to graduate more students from college. 
If students did as well in courses using very early MOOCs in spite of the many hurdles they and their 
professors had to overcome in this experiment, how might they fare using more advanced MOOCs, or 
MOOCs that were a better fit for their courses? Would faculty members take advantage of them if they 
had adequate rights, incentives, and support? These questions merit further exploration. 
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